Michael JohnsonSales teams hear the same question from different channels: what problems are potential customers already trying to solve? Searching TikTok manually produces a scatter of videos, but it is difficult to repeat the search a week later, compare results, and explain why a particular video made it into a report. For a small research project, you can turn public results for a few precise queries into a reproducible data pipeline: save the raw response, extract video and author fields, remove duplicates, and separately assess indicators of intent to buy or choose a product.
Below is a working Python workflow using the TikTok Scraper API from jsonscraper. For example, it could help a video-editing service see how creators describe difficulties with subtitles, templates, and exports. The script collects returned search results, then an analyst reviews the classification and decides which wording to use in audience research. This is a way to organize observations of public content, not an automatic count of all users who intend to buy.
The problem: audience queries exist, but research is hard to repeat
Manual research can easily lose its context. One employee searches for “how to add subtitles,” another for “automatic subtitles for Reels,” and a third saves only videos they like. The resulting spreadsheet mixes different queries, dates, metrics, and selection criteria. When a manager asks to repeat the analysis or compare two weeks, the team has to reconstruct not only the results but also the search process itself.
It is more useful to define the unit of observation in advance: a video found in search, with its description, identifier, link, author, and available public metrics. Save the original search query and collection time for each video. Then you can answer separate questions: which phrases appear, which problems are mentioned, which videos received more public reactions, and what changed between saved snapshots.
For this example, consider a company developing a tool for creating subtitles and short videos. It wants to find public videos in which creators discuss manual transcription, speech-recognition errors, subtitle timing, or preparing content for multiple platforms. Do not label a creator a “lead” or “buyer” based on a single video: a researcher can rank the content by observable indicators, then manually verify the conclusions.
Solution outline: search, record, clean, assess
The pipeline has four stages. First, the team creates a small set of search queries, each corresponding to a specific audience task. The script then requests video search and saves the full JSON response. The third stage normalizes different response structures into one row per video and removes duplicates by identifier. Finally, rules or a separate AI classifier assign a priority for manual review, while preserving the reason for the score.
Separating these stages is important for reproducibility. The raw response helps investigate changes to fields or normalization errors. The normalized table is convenient for filtering and visualization. A separate scoring table makes it possible to change the rules without requesting the API again, provided the saved data can be reprocessed that way and complies with the team’s retention period.
For a starting set of queries, try “automatic subtitle errors,” “how to add subtitles quickly,” and “editing short videos for business.” These are only examples; adapt the wording to the audience’s language and the product. Store queries alongside the results because different search terms produce different samples.
What the TikTok Scraper API provides
jsonscraper’s published documentation specifies the GET route /searchVideo at the base address https://tiktok.evelode.com for video search. The required parameter is keyword; documented optional parameters include count, cursor, region, date_posted, sort_by, and cache_timeout. For example, sort_by accepts the documented values most-liked, relevance, and date-posted, while date_posted accepts values such as this-week and all-time. The TikTok API endpoint reference lists the exact parameters and describes the route.
The key is passed as the license_key query parameter. In the example below, it is read from the JSONSCRAPER_TIKTOK_LICENSE_KEY environment variable so it is not stored in the code or printed in logs. Use the current terms of your access for authentication and parameter selection; the TikTok Scraper API page describes the product and links to a demo and documentation.
Parse search responses based on their structure rather than assuming there is only one array. In a verified example, the API returned a top-level status and a tiktok object; videos may be found in tiktok.aweme_list, while search results may also appear in tiktok.search_item_list[].aweme_info. Check both forms. The returned objects may include a video identifier, description, link, publication time, author information, and statistics such as views, likes, comments, and shares. Treat metrics and fields as optional: the code should not fill in made-up zeros when a value is missing.
Search pagination uses cursor. If the script continues through the results, it should pass the returned value as an opaque token for the next request rather than calculating one itself. For an initial prototype, it is reasonable to limit collection to one page per query: this simplifies verification and makes the workload predictable. Pagination can be added later, with a page limit and protection against repeated cursors.
Set up the environment
The example uses Python 3.8 or later and the standard library. Set the key in the environment, for example in a macOS or Linux shell:
export JSONSCRAPER_TIKTOK_LICENSE_KEY="your_key"
python tiktok_demand.pyIn Windows PowerShell, set the variable as follows:
$env:JSONSCRAPER_TIKTOK_LICENSE_KEY="your_key"
python .\tiktok_demand.pyThe script makes a separate request for each keyword. It checks for HTTP errors, valid JSON, and the top-level status, then extracts objects from both confirmed array structures. Raw responses are written one per line to a JSONL file: each record contains the query, observation time, and original response. The second part creates a CSV with normalized fields. The key is not included in the code, and the secret is never printed.
Example: collect and normalize search results
import csv
import json
import os
import time
from datetime import datetime, timezone
from pathlib import Path
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import Request, urlopen
BASE_URL = "https://tiktok.evelode.com/searchVideo"
QUERIES = [
"automatic subtitle errors",
"how to add subtitles quickly",
"editing short videos for business",
]
def extract_videos(payload):
"""Supports both observed search response structures."""
tiktok = payload.get("tiktok") or {}
videos = []
aweme_list = tiktok.get("aweme_list") or []
if isinstance(aweme_list, list):
videos.extend(item for item in aweme_list if isinstance(item, dict))
search_items = tiktok.get("search_item_list") or []
if isinstance(search_items, list):
for item in search_items:
if not isinstance(item, dict):
continue
video = item.get("aweme_info")
if isinstance(video, dict):
videos.append(video)
return videos
def fetch(query, license_key):
params = {
"keyword": query,
"count": 20,
"sort_by": "relevance",
"date_posted": "this-month",
"region": "US",
"license_key": license_key,
}
url = BASE_URL + "?" + urlencode(params)
request = Request(url, headers={"Accept": "application/json"})
try:
with urlopen(request, timeout=45) as response:
payload = json.loads(response.read().decode("utf-8"))
except HTTPError as exc:
raise RuntimeError("API returned HTTP status {}".format(exc.code)) from exc
except (URLError, TimeoutError, json.JSONDecodeError) as exc:
raise RuntimeError("Could not complete or parse API request") from exc
if payload.get("status") != "ok":
raise RuntimeError("API response status was not ok")
return payload
def metric(statistics, key):
value = statistics.get(key)
return value if isinstance(value, (int, float)) else ""
def main():
license_key = os.environ.get("JSONSCRAPER_TIKTOK_LICENSE_KEY")
if not license_key:
raise SystemExit("Set JSONSCRAPER_TIKTOK_LICENSE_KEY in the environment")
raw_path = Path("tiktok_raw.jsonl")
csv_path = Path("tiktok_videos.csv")
seen_ids = set()
rows = []
with raw_path.open("a", encoding="utf-8") as raw_file:
for query in QUERIES:
observed_at = datetime.now(timezone.utc).isoformat()
payload = fetch(query, license_key)
raw_file.write(json.dumps({
"query": query,
"observed_at": observed_at,
"response": payload,
}, ensure_ascii=False) + "\n")
for video in extract_videos(payload):
video_id = video.get("aweme_id")
if not video_id or video_id in seen_ids:
continue
seen_ids.add(video_id)
author = video.get("author") or {}
stats = video.get("statistics") or {}
rows.append({
"query": query,
"observed_at": observed_at,
"video_id": video_id,
"created_at": video.get("create_time", ""),
"description": video.get("desc", ""),
"share_url": video.get("share_url", ""),
"author_username": author.get("unique_id", ""),
"author_nickname": author.get("nickname", ""),
"views": metric(stats, "play_count"),
"likes": metric(stats, "digg_count"),
"comments": metric(stats, "comment_count"),
"shares": metric(stats, "share_count"),
})
time.sleep(1)
columns = [
"query", "observed_at", "video_id", "created_at", "description",
"share_url", "author_username", "author_nickname", "views",
"likes", "comments", "shares",
]
with csv_path.open("w", newline="", encoding="utf-8") as csv_file:
writer = csv.DictWriter(csv_file, fieldnames=columns)
writer.writeheader()
writer.writerows(rows)
print("Saved {} unique returned videos to {}".format(len(rows), csv_path))
if __name__ == "__main__":
main()The count value here specifies the requested sample size per query. The script does not assume that the API must return exactly that many items: the actual number of rows depends on the response. The pause between requests is a simple way to regulate the run rate, not a claim about a service limit or requirement. For a production schedule, add bounded retries, error monitoring, and rules for an appropriate request frequency based on your plan’s terms.
How to read the resulting table
Each CSV row represents one unique returned video found by one of the queries. The query field shows where the record came from, observed_at gives the snapshot time, and video_id serves as the deduplication key. The description and link help the analyst verify context, while public counters help assess the response at collection time. For long-term comparisons, use separate observations: do not overwrite metric history if you need to see how it changes.
View, like, comment, and share counts are snapshots of the public statistics returned by the API at the time of the request. They may be missing from individual objects, so a blank value in the CSV means that no numeric field was present in the response. This differs from zero and should be handled separately when calculating aggregates. In addition, the returned results reflect the search sample for specific terms, region, time filter, and request parameters. They describe the videos retrieved, not the platform’s full volume of content or audience.
For an initial assessment of potential relevance, you can build a transparent rules table: matches with problem descriptions, mentions of a tool in use, questions about price or comparisons, and specific work scenarios. It is better to store separate indicators and an explanation for the score than to record only an opaque number. For example, a video asking about the accuracy of automatic transcription could be marked “review: discusses quality,” while one with general content tips could be marked “low topical relevance.”
Use AI without making it the judge
Once you have a sample, a model can help label descriptions using a predefined schema: problem, tool mentioned, question sentiment, and whether there is a product-selection signal. Send the model only the fields it needs—for example, the description text and record identifier—and ask it to return fixed JSON with labels and a brief explanation. Save the prompt version, model, and result alongside the original row so that you can repeat the labeling or have a person spot-check it.
A useful classifier prompt should limit its conclusions to observable indicators. Ask it to identify whether the text contains a clear subtitle problem, a request for a tool recommendation, or discussion of a workflow. If there is not enough information, include an “insufficient context” category. The model should not infer the creator’s identity, purchasing intentions, or demographic characteristics from one public video. The result is a queue of topics to analyze, not a list of people to contact automatically.
To assess labeling quality, take a random subsample and manually assign the same categories. Compare agreement for each label and review the errors: for example, the model might mistake sarcasm for a complaint or an educational tip for personal experience. Improving the query vocabulary and instructions is often more useful than immediately making the model more complex. For reporting, retain original text and links in line with the agreed data retention and access policy.
Schedule automation and monitor quality
After a successful manual run, you can schedule the script with cron, a cloud platform’s task scheduler, or an orchestrator such as n8n. The practical sequence of steps remains the same: read the query list, call the API, check the status, save the raw response, normalize videos, record new snapshots, and generate a report. Pass the key through the runtime’s secrets mechanism, not in a spreadsheet, workflow text, or error message.
Four checks are useful in an automated task. First, the request completed successfully and the response is valid JSON. Second, the top-level status matches the expected value. Third, the handler understands both search structures and handles missing fields correctly. Fourth, the number of new unique videos and the number of errors are logged. If there are suddenly no results, the report should show an empty sample as the state of that run rather than present it as a normal successful report without explanation.
Add pagination only after checking the first page. The search route’s documentation specifies a cursor parameter; save the returned token and pass it unmodified in the next request. Limit the maximum number of pages per search term and stop the loop if there are no new items, no next cursor, or a cursor that has already been processed. This limit keeps run time and collection size manageable. The cursor lets you request more results, but does not establish that the results cover every post.
Repeated runs on the same day may return overlapping videos. Therefore, the deduplication key should be a stable video identifier, while snapshot history should be stored as a separate record with the observation date. If you need a consolidated view, build it on top of the snapshots: this lets you distinguish a video found for the first time from one that appeared in several queries or was retrieved again later.
Metrics: what you can compare reliably
Useful operational metrics include the number of unique videos found per query, the share of videos with a description, the distribution of available reactions, and the frequency of particular topics. When comparing queries, keep the parameters and run period consistent. If one run uses this-week and another uses this-month, the difference in volume may be due to the time window, so it cannot be interpreted as a change in interest without additional analysis.
Comparing a video’s metrics over time requires several snapshots and a consistent storage schema. Keep the original values as returned for each date, then calculate the difference between observations. Do not call the difference a one-day increase if the intervals between actual runs varied. Similarly, ranking by likes can help find prominent videos in the collected sample, but does not measure the creator’s or product’s commercial effectiveness.
A final report for the marketing team can combine three levels: examples of audience wording with clickable links, thematic groups of problems, and a quantitative description of the sample collected. Support each conclusion with the number of observations, queries, and collection period. This gives managers context and lets analysts repeat the filtering on the same source data.
Common mistakes and how to avoid them
Treating a missing field as zero
If statistics or author information are missing, the code above leaves the cell blank. This preserves the distinction between an unknown value and a measured zero. When analyzing the data, explicitly define how to handle missing values and report the number of rows excluded from each calculation.
Assuming search always returns a single array
Search responses may contain aweme_list and nested search_item_list[].aweme_info items. The handler checks both forms and ignores items of the wrong type. If the schema changes, saved raw responses will help identify the new location of the data quickly, while test JSON fixtures let you verify the fix before the next run.
Counting one video as a separate signal more than once
A video may match several search terms. In the example, it is included in the final CSV once, and the query field indicates the first match in the current order. If you need to retain all matches between videos and queries, store a separate linking table or a list of queries for each identifier rather than discarding the record’s origin.
Comparing samples that are not comparable
Search term, region, time filter, sorting, and run date affect the sample. Save the parameters alongside the response and change one factor at a time when comparing variants. This helps determine whether a change in results is related to search behavior or to a change in the experiment settings.
Presenting correlation as purchase intent
Video descriptions and public counters provide indicators of content and audience response. On their own, they do not show who is ready to buy, whether someone saw an ad, or what the conversion rate is. Use them to develop hypotheses, validate topics, and prepare content. Evaluate sales effectiveness using your own campaign data, aligned with this research by time and audience.
How to expand the pipeline
Once the basic version works reliably, you can add a configuration file with the query list and run parameters so you can change the vocabulary without editing Python. The next step is a mapping table for queries and topic clusters, such as “recognition quality,” “editing speed,” and “export.” The team should define these categories and check them against examples; the API returns search data, while the researcher defines the meaning-based schema.
A relational database may be a better storage option than CSV: use tables for runs, videos, and metric snapshots. A unique index on the video identifier prevents duplicate entries in the catalog, while snapshots let you track changes in available counters. Store observation dates in UTC and record the normalizer version. If the extraction logic changes, you can reprocess the archived JSONL and compare the result.
In n8n, you can schedule collection and run the normalization code in a separate Python service or node if the chosen environment supports the required operations. You do not need to integrate a model at the first stage: first make collection reliable, then measure how much time manual classification takes and which categories recur. AI is worthwhile when it reduces a specific repetitive task and its labels can be checked.
If your research question is about regularly monitoring a specific account rather than searching by wording, the official reference lists a separate /getUserFeed route with the parameters query, max_cursor, and count. For a hashtag, it provides /getHashtagFeed with the cursor parameter. These are different collection scenarios: choose the endpoint based on the research question rather than substituting one type of sample for another.
Result: a repeatable sample instead of a collection of bookmarks
After running the pipeline, the team gets two useful artifacts: an archive of raw responses and a table of unique videos with the search term, observation date, link, description, and available metrics. These can be used to group audience questions manually, pass text through controlled classification, create a weekly summary, and compare repeated snapshots. The code explicitly handles request errors, checks the top-level status, distinguishes two search structures, and preserves missing metrics as missing values.
The practical outcome for the team is a more transparent decision about which problems to investigate in interviews, landing pages, or ad copy. If the research requires a different type of data, change the queries or use the appropriate documented route. Before adapting the pipeline, check the TikTok API documentation; product information and the demo entry point are on the TikTok Scraper API page.
This approach addresses the original task: turning TikTok searches into a reproducible research process. The script collects specific returned videos, preserves their provenance and snapshots, and a person or verifiable classifier turns the content into hypotheses about demand. The team gets a basis for its next decision while maintaining the distinction between public content indicators and confirmed sales outcomes.
