Michael JohnsonWhen a team is looking for a topic for its next campaign, manually reviewing dozens of profiles quickly turns into a collection of notes with no consistent method. One analyst records views, another notes video captions, and posts with similar wording end up in different spreadsheets. A week later, it is hard to tell what is recurring most often: a topic, a presentation style, or simply a successful video.
A practical approach is to maintain a short list of public accounts, regularly retrieve information about their videos in a structured format, and save each collection as a separate snapshot. You can then compare posts and available metrics over time, using a language model to group captions as a first pass. Below is an outline for a research pipeline in Python using the jsonscraper TikTok Scraper API.
Define the question before collecting data
Imagine a marketing team at a small sports nutrition brand. It wants to find out which topics regularly appear among selected public TikTok creators and which videos merit manual review. The goal is to compare posts from preselected accounts over an observation period, not to build a universal ranking of TikTok popularity.
This sets the boundaries of the analysis. Collecting feeds from selected profiles helps examine the content and posting frequency of those accounts. Available public counters can be recorded at the time of the request. Evaluating the performance of your own advertising requires separate data—for example, spend and conversions from an ad platform or web analytics.
Before connecting to the API, document the research protocol: the account list, start date, collection frequency, fields to analyze, and criteria for including posts. Record the time of each request. This makes it possible to distinguish a change in a video's metrics from a difference between two sets of observations.
What to collect and what to measure
A small project needs three layers of data. The first is the account identifier and collection run time. The second is the original API response, saved for later verification. The third is normalized data for queries and reports: video ID, caption, publication time, and available engagement counters.
Determine field names and response nesting from an example of the specific endpoint. Do not assume in advance that the list is always called videos or that the view counter is views. First make a documented request and inspect the actual JSON, then configure the transformation to your own schema.
Useful metrics include the number of new posts per interval, the distribution of posts by topic, and changes in public counters between snapshots of the same video. For the last metric, match records using the same video ID. This keeps the performance of an already-published video separate from the metrics for new posts.
Connecting to the TikTok Scraper API
The TikTok Scraper API overview lists the base URL https://tiktok.evelode.com, and the endpoint reference confirms GET /getUserFeed and the parameters query, max_cursor, count, region, with_pinned_posts, sort_type, and cache_timeout. For your first request, specify a profile in query and, for example, the number of items using count. The license_key is passed in the query string; its configuration is also described in the API reference. Use the Postman collection to check the endpoint and inspect the response before integration.
Store the key in an environment variable. Do not add it to source code, a repository, a spreadsheet, or a client application. Make a request, save a successful JSON example with the date and endpoint name, and use it as the basis for a normalization adapter and tests.
A minimal working request in Python
The reference says that the user feed response contains an outer status and a tiktok object, with the post list at aweme_list. Records may include an ID, caption, publication time, author, and statistics. Check the actual fields and values in the response you receive: some data may be missing.
import json
import os
from datetime import datetime, timezone
from pathlib import Path
import requests
API_KEY = os.environ["JSONSCRAPER_TIKTOK_LICENSE_KEY"]
ENDPOINT_URL = "https://tiktok.evelode.com/getUserFeed"
def fetch_user_feed(handle: str, count: int = 20) -> dict:
response = requests.get(
ENDPOINT_URL,
params={
"license_key": API_KEY,
"query": handle,
"count": count,
"region": "US",
},
timeout=45,
)
response.raise_for_status()
payload = response.json()
if not isinstance(payload, dict):
raise ValueError("Expected a JSON object")
if payload.get("status") != "ok":
raise RuntimeError(f"API returned status: {payload.get('status')!r}")
tiktok = payload.get("tiktok")
if not isinstance(tiktok, dict):
raise ValueError("The response is missing the tiktok object")
return payload
def save_raw(payload: dict, account: str, output_dir: str = "data/raw") -> Path:
collected_at = datetime.now(timezone.utc).isoformat()
safe_account = "".join(c for c in account if c.isalnum() or c in "-_@")
path = Path(output_dir) / f"{safe_account}_{collected_at.replace(':', '-')}.json"
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(
json.dumps(
{"collected_at": collected_at, "account": account, "payload": payload},
ensure_ascii=False,
indent=2,
),
encoding="utf-8",
)
return path
def normalize_items(payload: dict, account: str, collected_at: str) -> list:
items = payload["tiktok"].get("aweme_list", [])
if not isinstance(items, list):
raise ValueError("Field tiktok.aweme_list is not a list")
observations = []
for item in items:
if not isinstance(item, dict):
continue
stats = item.get("statistics")
stats = stats if isinstance(stats, dict) else {}
author = item.get("author")
author = author if isinstance(author, dict) else {}
observations.append({
"account_handle": account,
"collected_at": collected_at,
"video_id": item.get("aweme_id"),
"caption": item.get("desc"),
"published_at_epoch": item.get("create_time"),
"author_handle": author.get("unique_id"),
"views": stats.get("play_count"),
"likes": stats.get("digg_count"),
"comments": stats.get("comment_count"),
"shares": stats.get("share_count"),
})
return observations
if __name__ == "__main__":
account = "creator_handle"
payload = fetch_user_feed(account, count=20)
collected_at = datetime.now(timezone.utc).isoformat()
raw_path = save_raw(payload, account)
observations = normalize_items(payload, account, collected_at)
print(f"Saved: {raw_path}; records: {len(observations)}")Before running the script, set JSONSCRAPER_TIKTOK_LICENSE_KEY in the environment and install the requests library. The code checks the HTTP response, outer status, and presence of the tiktok object, then reads aweme_list. It saves the raw JSON and extracts fields confirmed by the reference, leaving missing values empty. The feed is the set of results returned by the API, so reports should be tied to the request time and selected profile.
Store snapshots and compare posts
SQLite or PostgreSQL will work for a small project. Store the raw response separately from normalized records. You can use a schema like this for the observations table:
CREATE TABLE IF NOT EXISTS video_observations (
account_handle TEXT NOT NULL,
video_id TEXT NOT NULL,
collected_at TEXT NOT NULL,
published_at_epoch INTEGER,
caption TEXT,
views INTEGER,
likes INTEGER,
comments INTEGER,
shares INTEGER,
raw_file TEXT,
PRIMARY KEY (account_handle, video_id, collected_at)
);Each run adds a new snapshot instead of overwriting the previous one. When comparing counters, calculate the difference only when both values are available for the same video. Include the interval between requests and the observation time in the report: they explain the period over which the change accumulated.
For topic grouping, start with a small category dictionary, then add a classifier or language model if needed. Pass the model cleaned captions, post IDs, and predefined categories. A structured result containing a category and a short rationale is easier to compare across runs than an open-ended summary.
Add a language model with verifiable labels
A model can initially sort captions into topics such as “recipes,” “workouts,” “product reviews,” and “personal stories.” Start by manually labeling a few dozen posts. Compare the model's labels with these examples, refine the category definitions, and repeat the evaluation after changing the prompt or model.
Calculate the number of new posts, median counters, and differences between snapshots with ordinary code; these are more reliable that way. Use the model to classify text and prepare short explanations. Keep the original captions and numerical observations available for verification.
Store the post ID, prompt version, processing date, and model used alongside each label. If a classification is wrong, an analyst can find the caption and correct the rule. It is useful to store the original label and manual correction separately.
Automate the run
After checking a manual run, move the script to cron, GitHub Actions, n8n, or Make. The scheduler runs the collection on a schedule, passes in the account list, logs the result, and sends the report to the chosen storage destination. Store the key in the runtime environment's secrets. Test the process with one account first, then expand the list.
Split the automation into stages: account queue, API call, raw response storage, normalization, snapshot insertion, and report generation. An error with one account should not erase results for the others. You can make reruns idempotent using a run ID and database uniqueness constraints.
Log the time, account, endpoint, HTTP status, duration, number of items processed, and error description. Do not write the API key to the logs. The cache_timeout parameter is listed for /getUserFeed; add it to the request if you need a specified cache duration. The API reference describes the default as 3600 seconds.
Check quality before publishing a report
Compare several normalized records against the original JSON. Check that a repeat collection recognizes the same video by ID, dates are parsed as expected, numeric fields have the right type, and the proportion of missing values is known. Then run the process again and confirm that the database retained old observations and added new ones.
Do not mix accounts or collection periods. When comparing topics, use periods of equal length or show their lengths clearly in the report. For counter trends, record the interval between snapshots so readers understand the scale of the intervals being compared.
Keep the original caption text and create a cleaned version separately. Emojis, hashtags, links, and language-specific features can affect classification; separate fields make it easier to check the effects of cleaning and recalculate categories without another API request.
What the pipeline will produce
After several cycles, the team will have a table of posts and snapshots, an initial grouping of captions by topic, and a report on new videos over the selected period. This helps identify videos for expert review, spot recurring topics among selected accounts, and check how public counters changed between observation dates.
These data can help generate hypotheses for an editorial calendar: compare different ways of presenting the same question or identify topics to test in your own content. To assess commercial impact, compare the observations with internal campaign data.
To get started, choose a small list of public accounts and test GET /getUserFeed in the Postman request collection. Add only parameters and field paths to the code that are confirmed by the endpoint documentation and its actual response.
Responsible collection of public data
Limit the account list to the research objective, collect only the necessary fields, and set a retention period for raw responses. Before using the data commercially, check the platform's applicable terms and the personal data requirements in your jurisdiction. Access controls and deleting outdated snapshots help keep the process within scope.
Before sending data to an AI service, remove fields the model does not need. In the report, include the post ID and observation date so a team member can find the record and verify the basis for the conclusion.
Quick-start checklist
- Define the niche, account list, period, and analysis questions.
- Check the
GET /getUserFeedparameters in the reference and inspect the actual response in Postman. - Save a JSON example and configure the normalizer to use verified fields.
- Record the raw response and separate snapshots with their collection times.
- Calculate numerical metrics in code; use AI to group text as a first pass.
- Add scheduling, logging, retry handling, and a retention period.
This makes monitoring a reproducible research process: each conclusion is linked to a post and observation time, and the code relies on the actual schema of the selected endpoint.