jsonscraper
Markets
Back to journal

How to Turn TikTok Search into a Weekly Change Log

A Python script saves newly found videos, compares them by ID, and creates a CSV for regular topic reviews

Editorial Policy Report an error

A weekly TikTok search report is useful when it shows which videos appeared in the latest results since the previous run. To do this, save the IDs of videos already seen and add new results to a CSV. Below is a reproducible Python script using the TikTok Scraper API: it searches videos by keyword, parses both documented response formats, and maintains a local ID database.

The goal: spot new posts instead of reviewing the entire results list

A support team, market researcher, or content analyst may need a regular sample of public videos for a product-related query. Exporting the full results every time makes repeated videos harder to review and new posts easier to miss among those already processed. An ID log makes each run comparable: the CSV contains only videos new to this process, along with the main fields returned in the response.

This log answers a limited but practical question: which items appeared in the current run’s response but were absent from the saved set of IDs? This is useful for a manual review queue and for sending data to a spreadsheet or subsequent analysis. The sample reflects the API results for the specified query; it does not, by itself, describe every TikTok post.

What you need

  • Python 3.8 or later.
  • Access to the TikTok Scraper API and a license key.
  • A query you can repeat without changing its meaning, such as a product name or a specific phrase.

Save the key in the JSONSCRAPER_TIKTOK_LICENSE_KEY environment variable rather than in the script file. In the example below, the key is passed as the license_key query parameter, as specified by the API schema. Do not print it in execution logs.

export JSONSCRAPER_TIKTOK_LICENSE_KEY="your_key"
python -m pip install requests

In Windows PowerShell, you can set the variable for the current session like this: $env:JSONSCRAPER_TIKTOK_LICENSE_KEY="your_key". Do not add real secrets to Git or a shared CSV.

Integration: run a search and normalize the JSON

For video search, the documentation describes the GET route /searchVideo with the required keyword parameter. You can also set count, region, sort order, time filter, and a cursor for paging through results. This minimal example uses only a keyword, the number of items to request, and the US region. The TikTok API endpoint reference lists the exact parameter names.

The search response requires special attention to its structure: the aweme_list array may be empty, while video items may appear in search_item_list, with data nested under aweme_info. The code therefore checks both paths, combines the items, and removes duplicates using aweme_id. This prevents successfully parsed JSON from being mistakenly turned into an empty report.

import csv
import json
import os
from pathlib import Path
from datetime import datetime, timezone

import requests

BASE_URL = "https://tiktok.evelode.com/searchVideo"
KEYWORD = "product name"
REGION = "US"
COUNT = 30
STATE_FILE = Path("seen_video_ids.json")
OUTPUT_FILE = Path("new_tiktok_videos.csv")

license_key = os.environ.get("JSONSCRAPER_TIKTOK_LICENSE_KEY")
if not license_key:
    raise RuntimeError("Set JSONSCRAPER_TIKTOK_LICENSE_KEY")

response = requests.get(
    BASE_URL,
    params={
        "license_key": license_key,
        "keyword": KEYWORD,
        "count": COUNT,
        "region": REGION,
    },
    timeout=60,
)
response.raise_for_status()
data = response.json()

# The example response contains an outer status and a nested tiktok object.
if "status" in data and data["status"] != "ok":
    raise RuntimeError("API returned a status other than ok")

tiktok = data.get("tiktok") or {}
items = list(tiktok.get("aweme_list") or [])
for entry in tiktok.get("search_item_list") or []:
    info = entry.get("aweme_info")
    if isinstance(info, dict):
        items.append(info)

# Unique items in the current response, preserving order.
current = {}
for item in items:
    video_id = item.get("aweme_id")
    if video_id:
        current[str(video_id)] = item

if STATE_FILE.exists():
    seen = set(json.loads(STATE_FILE.read_text(encoding="utf-8")))
else:
    seen = set()

new_items = [(video_id, item) for video_id, item in current.items()
             if video_id not in seen]
run_time = datetime.now(timezone.utc).isoformat()

fields = ["collected_at_utc", "query", "aweme_id", "created_at",
          "author", "description", "share_url", "play_count",
          "digg_count", "comment_count", "share_count"]
write_header = not OUTPUT_FILE.exists() or OUTPUT_FILE.stat().st_size == 0
with OUTPUT_FILE.open("a", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=fields)
    if write_header:
        writer.writeheader()
    for video_id, item in new_items:
        author = item.get("author") or {}
        stats = item.get("statistics") or {}
        writer.writerow({
            "collected_at_utc": run_time,
            "query": KEYWORD,
            "aweme_id": video_id,
            "created_at": item.get("create_time"),
            "author": author.get("unique_id") or author.get("nickname"),
            "description": item.get("desc"),
            "share_url": item.get("share_url"),
            "play_count": stats.get("play_count"),
            "digg_count": stats.get("digg_count"),
            "comment_count": stats.get("comment_count"),
            "share_count": stats.get("share_count"),
        })

seen.update(current.keys())
STATE_FILE.write_text(json.dumps(sorted(seen)), encoding="utf-8")
print(f"Response: {len(current)} unique videos; new: {len(new_items)}")

The script writes missing JSON values as empty cells: fields and metrics may not be present in an individual record. The code does not replace a missing value with zero or treat zero as missing data. Collection time is recorded separately from publication time, which the API returns in the create_time field.

How change detection works

  1. The request returns a page of search results for the specified keyword.
  2. The normalizer collects records from aweme_list and search_item_list[].aweme_info.
  3. The script removes duplicates within the response using the video ID.
  4. IDs are compared with the local JSON log.
  5. New IDs are added to the CSV, then the full set of IDs seen is saved for the next run.
Article: How to Turn TikTok Search into a Weekly Change Log
Summary: Set up repeatable TikTok video collection for a keyword query: handle both API response formats, remove duplicates, and save only new records for a weekly report.
Image role: supporting illustration 2 — a different specific aspect
Section: How change detection works
Visual subject: Clear infographic in English showing five verified workflow stages with labels: Request / Two response arrays / Deduplication by awe
The diagram shows the path from search response to a CSV of new records and a JSON log for the next run.© jsonscraper · AI-generated illustration
Article: How to Turn TikTok Search into a Weekly Change Log
Summary: Set up repeatable TikTok video collection for a keyword query: handle both API response formats, remove duplicates, and save only new records for a weekly report.
Image role: cover — the central idea
Section: How change detection works
Visual subject: Editorial still life of a weekly data review: stacked transparent cards representing public short-video search results, one newly appearing card beside a compac
IDs new to the local log are separated from videos found again.© jsonscraper · AI-generated illustration

Visualizing this process is useful when handing the script to colleagues: the CSV is an incremental log, while the JSON file stores deduplication state. They can be stored separately from the code and backed up together. For a team or cloud-based run, replace the local JSON file with suitable persistent storage and use atomic state updates so concurrent runs do not overwrite one another.

Repeat runs and pagination

Schedule repeat runs using an operating system scheduler, a CI task, or an automation workflow. Do not change the keyword without recording its version: different queries produce different samples, and mixing them in a single log makes interpretation harder. The script shown here uses one query for simplicity; you can process a list of queries in a loop and save the query field for each row.

The API supports cursor for paging through search results. If you add pagination, pass the cursor from the previous response, store it as an opaque value, and stop paging when no further pages are needed. The count parameter sets the number of items to request, but the actual number of records should be determined from the response. Even multiple pages are still pages of search results, not a guarantee of full platform coverage.

This script deliberately does not automatically classify text or declare a “trend.” The CSV helps quickly identify newly found records; an analyst can then label them by topic, intent, or question type and review examples manually. If you connect a language model, pass it only the text fields it needs and ask it to return categories alongside the original ID so the result can be checked and linked to the record.

Checks and common errors

  • HTTP error. raise_for_status() stops execution for an HTTP status code outside the successful range. Check API availability, parameters, and account access before updating the state file.
  • API status is not ok. The script stops before writing results if the response includes a top-level status field with another value. This preserves the previous log and helps avoid mistaking an erroneous response for an empty result set.
  • Empty list in aweme_list. Check search_item_list too; the code handles both structures.
  • Videos repeat across runs. This is expected when search result pages overlap: videos are not added to the report again if their IDs are already saved.
  • A metric is missing. Its cell remains empty. In calculations, distinguish missing values from actual zeros.
  • State was lost. The script will treat all returned IDs as new. Back up seen_video_ids.json along with the CSV, or use a persistent database.

For automation, record diagnostic details without the key: run time, HTTP status code, API status, number of parsed records, and number of new IDs. This helps distinguish a lack of new videos from a failed request. You can archive the full raw response separately if you need to recheck the schema later; restrict access and retention time according to the team’s needs.

Result

After running the script, you have a CSV of videos with IDs new to the log, descriptions, links, and available public fields, along with JSON state for comparison during the next run. This result works as a compact intake queue for support, analytics, or a content team. You can review the API description and access options on the TikTok Scraper API page; check exact search parameters against the official endpoint reference.

People

No people listed for this article yet.

Keep readingHow to Find Repeated Product Questions on TikTok
Read the next article

Turn what you read into a working integration

Explore jsonscraper's social-data APIs, test requests and build your next workflow.

Explore APIs