Caching Up with Claude

Playing around with Prompt Caching
Claude
Prompts
Published

August 15, 2024

Anthropic recently announced a new feature called “prompt caching” for the Claude API. This feature allows users to store complex instructions or multiple examples in the API’s memory, making them readily available for repeated use. While caching prompts initially costs 25% more than standard API usage, subsequent uses of cached prompts are 90% cheaper, offering significant cost savings for repetitive tasks.

This new feature holds particular promise for social scientists, like myself, who frequently process large volumes of documents using consistent sets of instructions.

To test it out, I thought I would try prompt caching on my current project of extracting quotes from newspaper articles about protest events.

pip install --upgrade anthropic
Requirement already satisfied: anthropic in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (0.34.0)
Requirement already satisfied: anyio<5,>=3.5.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (4.4.0)
Requirement already satisfied: distro<2,>=1.7.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (1.9.0)
Requirement already satisfied: httpx<1,>=0.23.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (0.27.0)
Requirement already satisfied: jiter<1,>=0.4.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (0.5.0)
Requirement already satisfied: pydantic<3,>=1.9.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (2.8.2)
Requirement already satisfied: sniffio in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (1.3.1)
Requirement already satisfied: tokenizers>=0.13.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (0.19.1)
Requirement already satisfied: typing-extensions<5,>=4.7 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anthropic) (4.12.2)
Requirement already satisfied: idna>=2.8 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anyio<5,>=3.5.0->anthropic) (3.7)
Requirement already satisfied: exceptiongroup>=1.0.2 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from anyio<5,>=3.5.0->anthropic) (1.2.0)
Requirement already satisfied: certifi in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from httpx<1,>=0.23.0->anthropic) (2024.7.4)
Requirement already satisfied: httpcore==1.* in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from httpx<1,>=0.23.0->anthropic) (1.0.5)
Requirement already satisfied: h11<0.15,>=0.13 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from httpcore==1.*->httpx<1,>=0.23.0->anthropic) (0.14.0)
Requirement already satisfied: annotated-types>=0.4.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from pydantic<3,>=1.9.0->anthropic) (0.7.0)
Requirement already satisfied: pydantic-core==2.20.1 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from pydantic<3,>=1.9.0->anthropic) (2.20.1)
Requirement already satisfied: huggingface-hub<1.0,>=0.16.4 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from tokenizers>=0.13.0->anthropic) (0.23.4)
Requirement already satisfied: filelock in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (3.15.4)
Requirement already satisfied: fsspec>=2023.5.0 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (2024.6.1)
Requirement already satisfied: packaging>=20.9 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (24.1)
Requirement already satisfied: pyyaml>=5.1 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (6.0.1)
Requirement already satisfied: requests in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (2.32.3)
Requirement already satisfied: tqdm>=4.42.1 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (4.66.4)
Requirement already satisfied: charset-normalizer<4,>=2 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (3.3.2)
Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/nealcaren/anaconda3/envs/code/lib/python3.10/site-packages (from requests->huggingface-hub<1.0,>=0.16.4->tokenizers>=0.13.0->anthropic) (2.2.2)
Note: you may need to restart the kernel to use updated packages.

Here are my directions. They are 1,326 tokens long.

instructions = '''

# Analysis of Direct Quotes, Slogans, and Statements

You are an AI assitant tasked with constructing a Markdown table using information from a newspaper article.

When analyzing media coverage of protests, it's essential to systematically capture and categorize the key quotes, slogans, chants, and statements made by various participants and stakeholders. This process allows for a clearer understanding of the perspectives, motivations, and messages conveyed during the protest. Whether you’re studying climate demonstrations, social justice movements, or any other form of protest, organizing this information into a structured table can greatly enhance your analysis and presentation of the event.

The following instructions provide a step-by-step guide to building a comprehensive table that categorizes and classifies quotes, slogans, chants, and statements from protest-related articles. This table will help you differentiate between the voices of protesters, bystanders, targets, and law enforcement, and will allow you to clearly present the various messages and actions that define the protest.


1. **Identify the Content:** 
   - Start by carefully reading the article and identifying all the direct quotes, slogans, chants, and statements. Highlight or note each instance where someone is directly quoted or where slogans/chants are mentioned.

2. **Extract the Content:**
   - For each identified content, extract the exact wording used in the article. Ensure you capture the quote, slogan, chant, or statement accurately. Use only direct quotes, ignore paraphrasing or other indirect quotes.

3. **Determine the Source:**
   - Identify who said or wrote each quote, slogan, chant, or statement. This could be an individual (e.g., a government official), a group (e.g., a protest organization), or an entity (e.g., a company).

4. **Classify the Role:**
   - Determine the role of the person or group who made the statement. Roles might include:
     - **Protester:** The individual or group actively involved in the protest.
     - **Bystander:** An observer who is neither directly involved in the protest nor its target.
     - **Target:** A person or entity that the protest or action is aimed at.
     - **Elected Official:** A person who holds elected or appointed office..
     - **Law Enforcement:** Police, government officials, or other authorities responding to the protest.

5. **Classify the Type of Content:**
   - Determine whether the content is a:
     - **Quote:** A direct statement made by an individual or group.
     - **Statement:** A formal declaration or assertion made by a group or individual, often in a press release or public statement.
     - **Slogan:** A short and memorable phrase used by the protestors, often seen on banners or signs.
     - **Chant:** A rhythmic phrase repeated by a crowd, usually during a protest.

6. **Create the Table:**
   - Set up a table with the following columns:
     1. **Quote/Statement:** The exact wording of the quote, slogan, chant, or statement.
     2. **Who Said It:** The individual, group, or entity that made the statement or created the slogan/chant.
     3. **Description:** A brief description of who the person or group is, including their role or title.
     4. **Role:** Classify the person or group as a protester, bystander, target, or law enforcement.
     5. **Type:** Classify the content as a quote, statement, slogan, or chant.

7. **Populate the Table:**
   - Fill in the table with the information you’ve gathered. Ensure that each row corresponds to a specific quote, slogan, chant, or statement, and that all columns are accurately completed.

8. **Review for Accuracy:**
   - Double-check the table for any errors or omissions. Ensure that the content matches the source material and that the roles and types are correctly assigned.

9. **Present the Table:**
   - Once the table is complete, it can be presented or used for analysis, ensuring that it provides a clear and organized overview of the key content from the article.

### Example:

| **Quote/Statement**                                                                                           | **Who Said It**                | **Description**                                         | **Role**            | **Type**           |
|---------------------------------------------------------------------------------------------------------------|--------------------------------|---------------------------------------------------------|---------------------|--------------------|
| "These criminal actions are dangerous and stupid. These anarchists are risking not only their own lives, but are also endangering others. We have recommended tough prison sentences. And we obligate airports to secure their facilities significantly better." | Nancy Faeser                   | Interior Minister of Germany                            | Law Enforcement     | Quote              |
| "Oil kills"                                                                                                   | Letzte Generation (Last Generation) | Slogan on banners unfurled by activists                  | Protester           | Slogan             |
| "Sign the treaty"                                                                                             | Letzte Generation (Last Generation) | Slogan on banners unfurled by activists                  | Protester           | Slogan             |
| "What’s dangerous is your political failure which is dragging us ever deeper into a catastrophe … we’re talking about our existence. About the lives of billions of people." | Letzte Generation (Last Generation) | Statement from the group defending their protest         | Protester           | Statement          |
| "Today’s disruptions at several airports are a concerted act of criminal extortion. This is not a peaceful protest and it is not about supposedly higher goals. These are malicious interventions in air traffic and in the personal rights of every traveller who is unable to take their flight as planned." | Ralph Beisel                   | Managing Director of the Airport Association ADV         | Target              | Statement          |

This set of directions should help in systematically creating a similar table for any article or document.
'''

My two sample articles, covering different protests.

article = '''Thousands march in India to protest doctor's rape, murder
VOA News
Protesters condemn the rape and murder of a doctor inside a government hospital in Kolkata, India, Aug. 14, 2024. 
Thousands of women took to the streets in several cities across India Wednesday night, holding candles and signs that said "reclaim the night" to protest the rape and murder of a 31-year-old doctor in Kolkata last week.

Prime Minister Narendra Modi called for strict punishment of crimes against woman, speaking against "the atrocities committed against our mothers, daughters and sisters."

"There is outrage against this in the country," he said. "I can feel this outrage."

India's top police agency on Wednesday took over the investigation of the case, which prompted a strike by doctors calling for safer working conditions.

The woman's body was discovered Friday in a state-run hospital in Kolkata, West Bengal, where she was a resident doctor. An autopsy confirmed she was sexually assaulted and killed.

Authorities arrested a police volunteer in connection with the crime.

On Monday, thousands of doctors marched in Kolkata and across West Bengal to protest the killing. They demanded justice for the victim and improved security at hospitals.

The protest spread nationwide on Tuesday, disrupting hospital services in several Indian cities. More than 8,000 government doctors halted work in all hospital departments except emergency services.

In New Delhi, junior doctors protested outside a major government hospital, holding signs that read, "Doctors are not punching bags," according to Reuters.

Doctors in government hospitals across several states have suspended elective services indefinitely, calling for swift justice and better workplace safety.

Sexual violence against women is a serious issue in India. In 2022, there were nearly 90 reported rapes daily in the country of 1.4 billion people.

This case has drawn comparisons to the 2012 gang rape and murder of a young woman on a Delhi bus, which led to significant public outcry and reforms.

In response to the 2012 incident, the government introduced harsher penalties for rape, including the death penalty for repeat offenders, and established new sexual offenses such as stalking.

Officials who failed to register rape complaints also faced jail sentences.

Some information for this report was provided by Agence France-Presse and Reuters.

The Voice of America provides news and information in more than 40 languages to an estimated weekly audience of over 326 million people. Stories with the VOA News byline are the work of multiple VOA journalists and may contain information from wire service reports.'''
article2 = '''Anti-racism campaigners march again in Belfast
6 days ago
Lyndsey Telford and Eimear Flanagan
BBC News NI

BBC Crowds of campaigners lining the street in front of Belfast City Hall BBC
Crowds of campaigners lined the street in front of Belfast City Hall
Several thousand anti-racism protesters gathered in Belfast city centre on Saturday for another demonstration following a week of disorder and racist attacks.

The rally was organised by a collective of organisations, including the trade union movement, United Against Racism and End Deportations Belfast.

United Against Racism said the event was intended to be a "peaceful response to racist violence of recent days".

"These are our streets, and we will not concede them to the racist forces who want to divide us," said the campaign group's chair, Fiona Doran.
Marchers gathered in Writers' Square in Belfast with a "refugees welcome" flag
The demonstration began in Writers' Square at noon on Saturday
Those attending Saturday's rally gathered in Writers' Square at noon before marching to the front of Belfast City Hall.

Placards with 'Ulster says no to racism' and 'Ulster says yeo to diversity' were among those waved by campaigners.

“Today was an amazing celebration of all that is good about our diverse city,” Ms Doran said.

“Now more than ever, we are determined to make Belfast an anti-racist city that prides itself on diversity.

"We will continue to fight to overcome the racist policies of the government, which punishes migrants, asylum seekers and refugees for coming here, and demonises people from religious and ethnic minority backgrounds."

Organisers said about 15,000 people took part in the rally.
Attack on mosque

Shortly after the rally began, the Police Service of Northern Ireland (PSNI) gave an update on their response to the upsurge in racially-motivated hate crimes.

They confirmed they are investigating an attempted petrol bombing of a mosque in Newtownards, County Down.

The device failed to ignite when it was thrown at the mosque on Friday, but racist graffiti was daubed on the building.

To date, 32 people have been arrested for offences linked to the recent disorder and racist attacks.
'Reclaiming our streets'

Mairead Sullivan and her daughter Nora attending the rally
Mairead Sullivan and her daughter Nora joined the anti-racism protest
One protester taking part in Saturday's march said the event was about "reclaiming our streets".

As they marched down Royal Avenue, the crowds chanted "Whose streets? Our streets" and "Belfast is anti-racist".

Mairead Sullivan and her daughter Nora also attended the parade and said they would protest every week if they needed to.

"I always taught my kids to respect everybody and treat them how you'd want to be treated, and we have to put our words into actions," she told BBC News NI.

Mairead was born in Belfast but her family moved to the United States to escape the Troubles.

She returned 12 years ago with her own family and said she does not want to see a return to “the dark days”.

Nora said the large turnout made her "proud" of Belfast.

"I think its so important that we stand up and support each other and we're here for the other members of our community," she added.

"We're all people. We're all just trying to have the best life we can and its important that we're here and make our voices are heard."
Campaigners at Writer's Square holding banners and flags
United Against Racism helped to organise Saturday's march and rally
A number of politicians were also present at Saturday's rally.

Claire Hanna, South Belfast and Mid Down MP, said she hoped the large turnout will be "uplifting and comforting" for those who feel fearful in the community.

"This was people from every community, every part of this city, saying very clearly that they do not want people to be attacked and scapegoated," she said.

"If people are claiming to defend their communities, they need to hear that message loud and clear."

Sinn Féin assembly member Deirdre Hargey called for those responsible for the violence minorities have faced to stop and be held accountable.

"This is the city of emancipation, of civil rights, and we need to work with communities to ensure that we do build an inclusive society which doesn’t hate on other communities, particularly our minority communities."
PA Media A woman with dark hair and clothing clapsPA Media
Sinn Féin MLA Dierdre Hargey was among politicians at the United Against Racism rally
People Before Profit MLA Gerry Carroll said the rally was a "powerful demonstration" of diversity within the city and society.

"I hope that it sends a message of support to those people who have been the target of the last week and the last period of time," he said.
Second large protest within days

Saturday's rally was the second large gathering of anti-racism campaigners within two days in the centre of Belfast.

On Friday more than 1,000 people gathered outside Belfast City Hall to oppose an anti-immigration protest at the same location.

The anti-racism activists outnumbered the anti-immigration protesters by about five to one.

The PSNI ensured the opposing sides were kept apart and both groups dispersed peacefully after Friday's demonstrations.
'''

A function to analyze an article. The cache_control block ensures that the system instructions are stored. As far as I can tell, caches don’t have an ID number, but instead are identified by their content. After five minutes of non-use, they disappear.

import anthropic
from IPython.display import Markdown

client = anthropic.Anthropic()


def analyze_article(article):
    response = client.beta.prompt_caching.messages.create(
        model="claude-3-5-sonnet-20240620",
        max_tokens=2044,
        system=[
            {
                "type": "text",
                "text": instructions,
                "cache_control": {"type": "ephemeral"},
            },
        ],
        messages=[
            {
                "role": "user",
                "content": article,
            }
        ],
    )
    return response

Analyze the two articles. For now, I’m ignoring accuracy, although I think it’s pretty good.

a = analyze_article(article)
Markdown(a.content[0].text)
PromptCachingBetaUsage(cache_creation_input_tokens=1326, cache_read_input_tokens=0, input_tokens=579, output_tokens=211)

Here’s a Markdown table based on the article, focusing on direct quotes, slogans, and statements:

Quote/Statement Who Said It Description Role Type
“reclaim the night” Protesters Slogan on signs held by protesters Protester Slogan
“There is outrage against this in the country. I can feel this outrage.” Narendra Modi Prime Minister of India Elected Official Quote
“Doctors are not punching bags” Junior doctors Slogan on signs held by protesting doctors in New Delhi Protester Slogan

Note: The article doesn’t contain many direct quotes or slogans. Most of the information is presented as paraphrased statements or general descriptions of events. The table includes the few instances of direct quotes and slogans that were explicitly mentioned in the article.

b = analyze_article(article2)
Markdown(b.content[0].text)
PromptCachingBetaUsage(cache_creation_input_tokens=0, cache_read_input_tokens=1326, input_tokens=1217, output_tokens=752)

Here’s a Markdown table summarizing the direct quotes, slogans, and statements from the article:

Quote/Statement Who Said It Description Role Type
“These are our streets, and we will not concede them to the racist forces who want to divide us.” Fiona Doran Chair of United Against Racism Protester Quote
“Today was an amazing celebration of all that is good about our diverse city.” Fiona Doran Chair of United Against Racism Protester Quote
“Now more than ever, we are determined to make Belfast an anti-racist city that prides itself on diversity.” Fiona Doran Chair of United Against Racism Protester Quote
“We will continue to fight to overcome the racist policies of the government, which punishes migrants, asylum seekers and refugees for coming here, and demonises people from religious and ethnic minority backgrounds.” Fiona Doran Chair of United Against Racism Protester Quote
“Whose streets? Our streets” Protesters Chant during the march Protester Chant
“Belfast is anti-racist” Protesters Chant during the march Protester Chant
“I always taught my kids to respect everybody and treat them how you’d want to be treated, and we have to put our words into actions.” Mairead Sullivan Protester at the rally Protester Quote
“I think its so important that we stand up and support each other and we’re here for the other members of our community.” Nora Sullivan Protester at the rally Protester Quote
“We’re all people. We’re all just trying to have the best life we can and its important that we’re here and make our voices are heard.” Nora Sullivan Protester at the rally Protester Quote
“This was people from every community, every part of this city, saying very clearly that they do not want people to be attacked and scapegoated.” Claire Hanna South Belfast and Mid Down MP Elected Official Quote
“If people are claiming to defend their communities, they need to hear that message loud and clear.” Claire Hanna South Belfast and Mid Down MP Elected Official Quote
“This is the city of emancipation, of civil rights, and we need to work with communities to ensure that we do build an inclusive society which doesn’t hate on other communities, particularly our minority communities.” Deirdre Hargey Sinn Féin assembly member Elected Official Quote
“I hope that it sends a message of support to those people who have been the target of the last week and the last period of time.” Gerry Carroll People Before Profit MLA Elected Official Quote
“Ulster says no to racism” Protesters Slogan on placards Protester Slogan
“Ulster says yeo to diversity” Protesters Slogan on placards Protester Slogan
“Refugees welcome” Protesters Flag at the rally Protester Slogan

Works great. I had Claude figure out how much this saves me, figuring the average article length would be 1,000 tokens and the average returned table would be 500 tokens.

# Constants
BASE_INPUT_PRICE = 3 / 1000000  # $3 per million tokens
CACHE_WRITE_PRICE = 3.75 / 1000000  # $3.75 per million tokens
CACHE_HIT_PRICE = 0.30 / 1000000  # $0.30 per million tokens
OUTPUT_PRICE = 15 / 1000000  # $15 per million tokens

CACHE_SIZE = 1326
REGULAR_INPUT_SIZE = 1000
REGULAR_OUTPUT_SIZE = 500

def calculate_costs_and_savings(num_calls):
    # Calculate cost without cache
    regular_cost = (BASE_INPUT_PRICE * (CACHE_SIZE + REGULAR_INPUT_SIZE) * num_calls) + (OUTPUT_PRICE * (CACHE_SIZE + REGULAR_OUTPUT_SIZE) * num_calls)
    
    # Calculate cost with cache
    cache_cost = (CACHE_WRITE_PRICE * CACHE_SIZE)  # First call writes to cache
    cache_cost += (CACHE_HIT_PRICE * CACHE_SIZE * (num_calls - 1))  # Subsequent calls use cache
    cache_cost += (BASE_INPUT_PRICE * REGULAR_INPUT_SIZE * num_calls)  # Regular input for all calls
    cache_cost += (OUTPUT_PRICE * (CACHE_SIZE + REGULAR_OUTPUT_SIZE) * num_calls)  # Output cost
    
    # Calculate savings
    savings = regular_cost - cache_cost
    
    return regular_cost, cache_cost, savings

# Calculate costs and savings for different number of calls
call_counts = [10, 100, 1000]

for calls in call_counts:
    regular_cost, cache_cost, savings = calculate_costs_and_savings(calls)
    print(f"For {calls} calls:")
    print(f"  Total cost without cache: ${regular_cost:.2f}")
    print(f"  Total cost with cache: ${cache_cost:.2f}")
    print(f"  Total savings: ${savings:.2f}")
    print()
For 10 calls:
  Total cost without cache: $0.34
  Total cost with cache: $0.31
  Total savings: $0.03

For 100 calls:
  Total cost without cache: $3.44
  Total cost with cache: $3.08
  Total savings: $0.35

For 1000 calls:
  Total cost without cache: $34.37
  Total cost with cache: $30.79
  Total savings: $3.58

Since the output is the expsenive part, you don’t save much here, but it does reduce the total cost by about 10% by only adding one line of code.