Update stats gathering functions by jimchamp · Pull Request #11844 · internetarchive/openlibrary

jimchamp · 2026-02-10T02:19:34Z

Supports #11714

Affects /stats page

Makes the following updates to our stats gathering functions:

Removes unimplemented functions
Updates comments that are stale or otherwise misleading
Adds new function that fetches all thing IDs associated with bot accounts
Simplifies functions that query for human and bot edit counts
Functions that query for bot/human edits now strictly count edit transactions (not records edited)

Important Note: Functions that count the number of edits on the platform continue to count every transaction type as an edit. To fix this, we'll have to filter by action type when querying the transaction table. I'm unsure if this filtering should be included in this PR. Some transactions are being misidentified today -- perhaps those should be fixed before filters are added to the queries.

Technical

Testing

Can be tested by running scripts/store_counts.py with the expected arguments. Before running, it's best to update the following values in admin/stats.py::store_data:

openlibrary/openlibrary/admin/stats.py

Line 62 in b83bdda

uid = "counts-%s" % date

Change to something like "test-counts-%s" % date to avoid clobbering existing data.

openlibrary/openlibrary/admin/stats.py

Line 66 in b83bdda

doc['type'] = 'admin-stats'

Change to something like "test-admin-stats" for easy clean-up after testing. With this change in place, you can delete the entries like this:

test_entry_keys = web.ctx.site.store.keys(type="test-admin-stats")
for key in test_entry_keys:
    web.ctx.site.store.delete(key)

The store entry created during testing can be compared to the most recent production store_counts entry. Expect the total_* counts to be different -- human edits may be off a fair amount, while bot edits may differ slightly.

Screenshot

Stakeholders

Removes all functions from `numbers.py` that return `0`. As all `admin_delta__*` functions have been removed, the code that calls these functions has been removed from `admin/stats.py`.

jimchamp · 2026-02-10T22:20:49Z

-    total_edits = result[0].count
+    bot_ids = _get_cached_bot_accounts(thingdb=kargs['thingdb'])
    q1 = (
-        "SELECT count(DISTINCT t.id) AS count FROM transaction t, version v WHERE "


Note that this was counting distinct transaction IDs. The bot edits query uses count(*), which would count the same transaction ID multiple times.

This should now be moot, as the version table is no longer queried for these counts.

Copilot

Pull request overview

This PR refactors the stats gathering functions in the admin module to improve accuracy and maintainability. It removes unimplemented stub functions, updates misleading documentation, adds bot account identification functionality, and simplifies the human/bot edit counting queries. The PR is marked as "DO NOT MERGE" and is a draft because it contains test values that must be replaced before production deployment.

Changes:

Removes unimplemented admin_delta__ functions and their corresponding gathering logic
Adds new get_bot_accounts() function to fetch bot account IDs from the database
Refactors admin_range__human_edits() and admin_range__bot_edits() to use the new bot account fetching mechanism
Updates documentation to fix typos and clarify function behavior

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
openlibrary/admin/stats.py	Adds test prefixes to data storage (test-counts and test-admin-stats) and removes admin_delta__ function calls
openlibrary/admin/numbers.py	Adds bot account fetching functions, refactors human/bot edit counting, removes unimplemented functions, and updates documentation

Removes notion of querying for bot accounts via the defunct `account` table. New functions that query the store for bot accounts have been created. `admin_range__human_edits` and `admin_range__bot_edits` have been updated to fetch bot accounts using the new functions. Additionally, a redundant DB query has been removed from `admin_range__human_edits`.

for more information, see https://pre-commit.ci

jimchamp · 2026-02-25T17:30:27Z

Data comparison

data generated with this updated script:

{"total_authors": 15069255, "total_covers": 14851352, "total_editions": 55613409, "total_lists": 253670, "total_members": 13382371, "total_works": 40792246, "type": "test-admin-stats", "authors": 1296, "bot_edits": 44791, "covers": 256, "editions": 3530, "human_edits": 7741, "lists": 118, "loans": 10382, "members": 4657, "works": 3050}

data generated with prod script:

{"total_authors": 15069258, "total_covers": 14851354, "total_ebooks": 0, "total_editions": 55613073, "total_lists": 253672, "total_members": 13382514, "total_subjects": 0, "total_works": 40791925, "ebooks": 0, "subjects": 0, "type": "admin-stats", "authors": 1292, "bot_edits": 0, "covers": 253, "editions": 3520, "human_edits": 51386, "lists": 115, "loans": 10255, "members": 4612, "works": 3041}

jimchamp · 2026-02-25T21:27:40Z

Noting that it took ~1 hour and 10 minutes to run this updated script for the previous 7 day's worth of counts.

* Remove unimplemented stats functions Removes all functions from `numbers.py` that return `0`. As all `admin_delta__*` functions have been removed, the code that calls these functions has been removed from `admin/stats.py`. * Updates functions which count bot and human edits Removes notion of querying for bot accounts via the defunct `account` table. New functions that query the store for bot accounts have been created. `admin_range__human_edits` and `admin_range__bot_edits` have been updated to fetch bot accounts using the new functions. Additionally, a redundant DB query has been removed from `admin_range__human_edits`.

jimchamp added 2 commits February 9, 2026 17:19

Remove unimplemented stats functions

ebaa039

Removes all functions from `numbers.py` that return `0`. As all `admin_delta__*` functions have been removed, the code that calls these functions has been removed from `admin/stats.py`.

Update misleading comments

4c1344e

jimchamp added the Needs: Testing label Feb 10, 2026

jimchamp commented Feb 10, 2026

View reviewed changes

jimchamp changed the title ~~Update stats gathering functions~~ [DO NOT MERGE] Update stats gathering functions Feb 18, 2026

jimchamp force-pushed the update-stats branch from 93aee05 to bd32120 Compare February 18, 2026 23:01

jimchamp commented Feb 23, 2026

View reviewed changes

Comment thread openlibrary/admin/numbers.py Outdated

jimchamp commented Feb 23, 2026

View reviewed changes

Comment thread openlibrary/admin/numbers.py Outdated

jimchamp marked this pull request as ready for review February 24, 2026 01:08

Copilot AI review requested due to automatic review settings February 24, 2026 01:08

jimchamp marked this pull request as draft February 24, 2026 01:09

Copilot started reviewing on behalf of jimchamp February 24, 2026 01:09 View session

jimchamp commented Feb 24, 2026

View reviewed changes

Comment thread openlibrary/admin/numbers.py Outdated

Copilot AI reviewed Feb 24, 2026

View reviewed changes

jimchamp force-pushed the update-stats branch from 37e1986 to 0b18821 Compare February 24, 2026 01:17

jimchamp marked this pull request as ready for review February 24, 2026 01:18

jimchamp force-pushed the update-stats branch from 6534891 to 37b31d7 Compare February 24, 2026 01:24

[pre-commit.ci] auto fixes from pre-commit.com hooks

380b713

for more information, see https://pre-commit.ci

RayBB added the On Testing label Feb 25, 2026

jimchamp changed the title ~~[DO NOT MERGE] Update stats gathering functions~~ Update stats gathering functions Feb 25, 2026

mekarpeles self-assigned this Mar 2, 2026

jimchamp removed the Needs: Testing label Mar 5, 2026

mekarpeles added the Priority: 1 Do this week, receiving emails, time sensitive, . [managed] label Mar 20, 2026

mekarpeles added this to the Sprint 2026-03 milestone Mar 23, 2026

cdrini removed the On Testing label Mar 31, 2026

mekarpeles added Priority: 0 Fix now: Issue prevents users from using the site or active data corruption. [managed] and removed Priority: 1 Do this week, receiving emails, time sensitive, . [managed] labels Apr 3, 2026

mekarpeles merged commit 42afb76 into internetarchive:master Apr 15, 2026
4 checks passed

jimchamp deleted the update-stats branch April 23, 2026 17:09

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Update stats gathering functions#11844

Update stats gathering functions#11844
mekarpeles merged 4 commits into
internetarchive:masterfrom
jimchamp:update-stats

jimchamp commented Feb 10, 2026 •

edited

Loading

Uh oh!

jimchamp Feb 10, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimchamp commented Feb 25, 2026 •

edited

Loading

Uh oh!

jimchamp commented Feb 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Uh oh!

Uh oh!

Conversation

jimchamp commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Technical

Testing

Screenshot

Stakeholders

Uh oh!

jimchamp Feb 10, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimchamp commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jimchamp commented Feb 25, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

jimchamp commented Feb 10, 2026 •

edited

Loading

jimchamp commented Feb 25, 2026 •

edited

Loading