Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
22d2a3b
Fix: remove duplicate stats from stats-usage
lohanidamodar Mar 5, 2025
59f9ed8
remove check
lohanidamodar Mar 5, 2025
706723e
reduce stats_resources interval for development
lohanidamodar Mar 5, 2025
1397bfd
Feat-batch writes in stats usage dump
lohanidamodar Mar 5, 2025
e1a6046
update stats resources interval in dev to 15s
lohanidamodar Mar 5, 2025
23c6f4b
improve test
lohanidamodar Mar 5, 2025
466cad2
dbForLogs resource
lohanidamodar Mar 5, 2025
3f2b89e
fetch relevant database metrics for logsDB
lohanidamodar Mar 5, 2025
1f870ab
Fix db usage endpoint
lohanidamodar Mar 5, 2025
1384834
Fix functions usage endpoint
lohanidamodar Mar 5, 2025
dfe1df7
Fix get storage usage endpoint
lohanidamodar Mar 5, 2025
655cb3f
Fix: get project usage endpoint
lohanidamodar Mar 5, 2025
45924d4
Merge branch '1.6.x' into fix-remove-duplicate-metrics-from-stats-usage
lohanidamodar Mar 5, 2025
85011ce
Merge branch 'feat-batch-writes-usage-dump' into fix-remove-duplicate…
lohanidamodar Mar 5, 2025
077aa32
use actual size
lohanidamodar Mar 5, 2025
f44dd0f
Fix database metrics
lohanidamodar Mar 5, 2025
6dafc67
Fix response
lohanidamodar Mar 5, 2025
d4bea35
add actual size in response model
lohanidamodar Mar 5, 2025
65ef126
fix
lohanidamodar Mar 5, 2025
1f27fc2
increase wait time to fix test
lohanidamodar Mar 5, 2025
139b414
Merge branch '1.6.x' into fix-remove-duplicate-metrics-from-stats-usage
lohanidamodar Mar 6, 2025
6ff6a7e
remove unused param
lohanidamodar Mar 6, 2025
918b4c3
use getDocument where relevant
lohanidamodar Mar 6, 2025
ef2fe4a
Fix id
lohanidamodar Mar 6, 2025
02fa2bd
fix typo
lohanidamodar Mar 6, 2025
e570dda
Merge branch '1.6.x' into fix-remove-duplicate-metrics-from-stats-usage
lohanidamodar Mar 6, 2025
0b1a218
improve stats fetching from db
lohanidamodar Mar 6, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .env
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,7 @@ _APP_MAINTENANCE_RETENTION_EXECUTION=1209600
_APP_MAINTENANCE_RETENTION_ABUSE=86400
_APP_MAINTENANCE_RETENTION_AUDIT=1209600
_APP_USAGE_AGGREGATION_INTERVAL=30
_APP_STATS_RESOURCES_INTERVAL=3600
_APP_STATS_RESOURCES_INTERVAL=15

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 The default stats-resources polling interval was changed from 3600 seconds (1 hour) to 15 seconds in the committed .env. This file is the shipped default for production deployments, and at 15 s the StatsResources worker will continuously count all documents in every bucket, function, and database — a very expensive set of queries — 240× more often than before. The CI workflow already overrides this via export _APP_STATS_RESOURCES_INTERVAL=5, so the .env value is not needed for tests to pass.

Suggested change
_APP_STATS_RESOURCES_INTERVAL=15
_APP_STATS_RESOURCES_INTERVAL=3600

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Performance Critical

_APP_STATS_RESOURCES_INTERVAL was changed from 3600 (1 hour) to 15 (15 seconds). This is clearly a debug value that was not reverted before merging. In production, the StatsResources worker will now run every 15 seconds instead of every hour, causing a massive and continuous load on every project database. The CI workflow correctly overrides this to 5 for tests, but that override does not affect production deployments which read from this file.

Suggested change
_APP_STATS_RESOURCES_INTERVAL=15
_APP_STATS_RESOURCES_INTERVAL=3600
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert ini developer with deep knowledge of security, performance, and best practices.

### Context

File: .env
Lines: 90-90
Issue Type: performance-critical
Severity: critical

Issue Description:
`_APP_STATS_RESOURCES_INTERVAL` was changed from `3600` (1 hour) to `15` (15 seconds). This is clearly a debug value that was not reverted before merging. In production, the `StatsResources` worker will now run every 15 seconds instead of every hour, causing a massive and continuous load on every project database. The CI workflow correctly overrides this to `5` for tests, but that override does not affect production deployments which read from this file.

Current Code:
_APP_STATS_RESOURCES_INTERVAL=15

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow ini best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue

_APP_MAINTENANCE_RETENTION_USAGE_HOURLY=8640000

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functional High

_APP_STATS_RESOURCES_INTERVAL was changed from 3600 (1 hour) to 15 seconds in the default .env shipped with the repository. The CI workflow correctly overrides this to 5 for tests, but any deployment that sources the bundled .env as a base will run the resource-counting worker every 15 seconds, causing severe unnecessary database load in production. The original value of 3600 should be restored.

Suggested change
_APP_MAINTENANCE_RETENTION_USAGE_HOURLY=8640000
_APP_STATS_RESOURCES_INTERVAL=3600
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert bash developer with deep knowledge of security, performance, and best practices.

### Context

File: .env
Lines: 91-91
Issue Type: functional-high
Severity: high

Issue Description:
`_APP_STATS_RESOURCES_INTERVAL` was changed from `3600` (1 hour) to `15` seconds in the default `.env` shipped with the repository. The CI workflow correctly overrides this to `5` for tests, but any deployment that sources the bundled `.env` as a base will run the resource-counting worker every 15 seconds, causing severe unnecessary database load in production. The original value of `3600` should be restored.

Current Code:
_APP_STATS_RESOURCES_INTERVAL=15

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow bash best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue

_APP_MAINTENANCE_RETENTION_SCHEDULES=86400
_APP_USAGE_STATS=enabled
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -163,6 +163,7 @@ jobs:

- name: Load and Start Appwrite
run: |
export _APP_STATS_RESOURCES_INTERVAL=5
docker load --input /tmp/${{ env.IMAGE }}.tar
docker compose up -d
sleep 30
Expand Down
148 changes: 92 additions & 56 deletions app/controllers/api/databases.php
Original file line number Diff line number Diff line change
Expand Up @@ -534,7 +534,6 @@ function updateAttribute(
}

$queueForEvents->setParam('databaseId', $database->getId());
$queueForStatsUsage->addMetric(str_replace(['{databaseInternalId}'], [$database->getInternalId()], METRIC_DATABASE_ID_STORAGE), 1); // per database

$response
->setStatusCode(Response::STATUS_CODE_CREATED)
Expand Down Expand Up @@ -830,9 +829,6 @@ function updateAttribute(
->setParam('databaseId', $database->getId())
->setPayload($response->output($database, Response::MODEL_DATABASE));

$queueForStatsUsage
->addMetric(METRIC_DATABASES_STORAGE, 1); // Global, deletion forces full recalculation

$response->noContent();
});

Expand Down Expand Up @@ -2732,9 +2728,6 @@ function updateAttribute(
->setContext('database', $db)
->setPayload($response->output($attribute, $model));

$queueForStatsUsage
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$db->getInternalId(), $collection->getInternalId()], METRIC_DATABASE_ID_COLLECTION_ID_STORAGE), 1); // per collection

$response->noContent();
});

Expand Down Expand Up @@ -3356,8 +3349,7 @@ function updateAttribute(

$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, $operations)
->addMetric(str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_OPERATIONS_WRITES), $operations)
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$database->getInternalId(), $collection->getInternalId()], METRIC_DATABASE_ID_COLLECTION_ID_STORAGE), 1); // per collection
->addMetric(str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_OPERATIONS_WRITES), $operations);

$response->addHeader('X-Debug-Operations', $operations);

Expand Down Expand Up @@ -4141,8 +4133,7 @@ function updateAttribute(

$queueForStatsUsage
->addMetric(METRIC_DATABASES_OPERATIONS_WRITES, 1)
->addMetric(str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_OPERATIONS_WRITES), 1)
->addMetric(str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$database->getInternalId(), $collection->getInternalId()], METRIC_DATABASE_ID_COLLECTION_ID_STORAGE), 1); // per collection
->addMetric(str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_OPERATIONS_WRITES), 1);

$response->addHeader('X-Debug-Operations', 1);

Expand Down Expand Up @@ -4186,28 +4177,51 @@ function updateAttribute(
->param('range', '30d', new WhiteList(['24h', '30d', '90d'], true), '`Date range.', true)
->inject('response')
->inject('dbForProject')
->action(function (string $range, Response $response, Database $dbForProject) {
->inject('dbForLogs')
->action(function (string $range, Response $response, Database $dbForProject, Database $dbForLogs) {

$periods = Config::getParam('usage', []);
$stats = $usage = [];
$days = $periods[$range];
$metrics = [
$logsDBMetrics = [
METRIC_DATABASES,
METRIC_COLLECTIONS,
METRIC_DOCUMENTS,
METRIC_DATABASES_STORAGE,
];

Authorization::skip(function () use ($dbForLogs, $days, $logsDBMetrics, &$stats) {
foreach ($logsDBMetrics as $metric) {
$result = $dbForLogs->getDocument('stats', md5('_inf_' . $metric));
$stats[$metric]['total'] = $result->getAttribute('value', 0);

$limit = $days['limit'];
$period = $days['period'];
$results = $dbForLogs->find('stats', [
Query::equal('metric', [$metric]),
Query::equal('period', [$period]),
Query::limit($limit),
Query::orderDesc('time'),
]);
$stats[$metric]['data'] = [];
foreach ($results as $result) {
$stats[$metric]['data'][$result->getAttribute('time')] = [
'value' => $result->getAttribute('value'),
];
}
}
});

$metrics = [
METRIC_DATABASES_OPERATIONS_READS,
METRIC_DATABASES_OPERATIONS_WRITES,
];

Authorization::skip(function () use ($dbForProject, $days, $metrics, &$stats) {
foreach ($metrics as $metric) {
$result = $dbForProject->findOne('stats', [
Query::equal('metric', [$metric]),
Query::equal('period', ['inf'])
]);

$result = $dbForProject->getDocument('stats', md5('_inf_'. $metric));
$stats[$metric]['total'] = $result['value'] ?? 0;

$limit = $days['limit'];
$period = $days['period'];
$results = $dbForProject->find('stats', [
Expand All @@ -4230,7 +4244,7 @@ function updateAttribute(
'1d' => 'Y-m-d\T00:00:00.000P',
};

foreach ($metrics as $metric) {
foreach ([...$logsDBMetrics,...$metrics] as $metric) {
$usage[$metric]['total'] = $stats[$metric]['total'];
$usage[$metric]['data'] = [];
$leap = time() - ($days['limit'] * $days['factor']);
Expand All @@ -4245,18 +4259,18 @@ function updateAttribute(
}
$response->dynamic(new Document([
'range' => $range,
'databasesTotal' => $usage[$metrics[0]]['total'],
'collectionsTotal' => $usage[$metrics[1]]['total'],
'documentsTotal' => $usage[$metrics[2]]['total'],
'storageTotal' => $usage[$metrics[3]]['total'],
'databasesReadsTotal' => $usage[$metrics[4]]['total'],
'databasesWritesTotal' => $usage[$metrics[5]]['total'],
'databases' => $usage[$metrics[0]]['data'],
'collections' => $usage[$metrics[1]]['data'],
'documents' => $usage[$metrics[2]]['data'],
'storage' => $usage[$metrics[3]]['data'],
'databasesReads' => $usage[$metrics[4]]['data'],
'databasesWrites' => $usage[$metrics[5]]['data'],
'databasesTotal' => $usage[$logsDBMetrics[0]]['total'],
'collectionsTotal' => $usage[$logsDBMetrics[1]]['total'],
'documentsTotal' => $usage[$logsDBMetrics[2]]['total'],
'storageTotal' => $usage[$logsDBMetrics[3]]['total'],
'databasesReadsTotal' => $usage[$metrics[0]]['total'],
'databasesWritesTotal' => $usage[$metrics[1]]['total'],
'databases' => $usage[$logsDBMetrics[0]]['data'],
'collections' => $usage[$logsDBMetrics[1]]['data'],
'documents' => $usage[$logsDBMetrics[2]]['data'],
'storage' => $usage[$logsDBMetrics[3]]['data'],
'databasesReads' => $usage[$metrics[0]]['data'],
'databasesWrites' => $usage[$metrics[1]]['data'],
]), Response::MODEL_USAGE_DATABASES);
});

Expand All @@ -4282,7 +4296,8 @@ function updateAttribute(
->param('range', '30d', new WhiteList(['24h', '30d', '90d'], true), '`Date range.', true)
->inject('response')
->inject('dbForProject')
->action(function (string $databaseId, string $range, Response $response, Database $dbForProject) {
->inject('dbForLogs')
->action(function (string $databaseId, string $range, Response $response, Database $dbForProject, Database $dbForLogs) {

$database = $dbForProject->getDocument('databases', $databaseId);

Expand All @@ -4293,20 +4308,43 @@ function updateAttribute(
$periods = Config::getParam('usage', []);
$stats = $usage = [];
$days = $periods[$range];
$metrics = [

$logsDBMetrics = [
str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_COLLECTIONS),
str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_DOCUMENTS),
str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_STORAGE),
];

Authorization::skip(function () use ($dbForLogs, $days, $logsDBMetrics, &$stats) {
foreach ($logsDBMetrics as $metric) {
$result = $dbForLogs->getDocument('stats', md5('_inf_' . $metric));
$stats[$metric]['total'] = $result->getAttribute('value', 0);

$limit = $days['limit'];
$period = $days['period'];
$results = $dbForLogs->find('stats', [
Query::equal('metric', [$metric]),
Query::equal('period', [$period]),
Query::limit($limit),
Query::orderDesc('time'),
]);
$stats[$metric]['data'] = [];
foreach ($results as $result) {
$stats[$metric]['data'][$result->getAttribute('time')] = [
'value' => $result->getAttribute('value'),
];
}
}
});

$metrics = [
str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASES_OPERATIONS_READS),
str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASES_OPERATIONS_WRITES)
Comment on lines +4340 to 4342

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functional High

The per-database usage metrics use the global metric names instead of the database-scoped metric constants, so database-specific read and write totals will always be empty; use the METRIC_DATABASE_ID_* templates here.

Suggested fix
        $metrics = [
            str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_OPERATIONS_READS),
            str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASE_ID_OPERATIONS_WRITES)
        ];
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert php developer with deep knowledge of security, performance, and best practices.

### Context

File: app/controllers/api/databases.php
Lines: 4340-4342
Issue Type: functional-high
Severity: high

Issue Description:
The per-database usage metrics use the global metric names instead of the database-scoped metric constants, so database-specific read and write totals will always be empty; use the METRIC_DATABASE_ID_* templates here.

Current Code:
        $metrics = [
            str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASES_OPERATIONS_READS),
            str_replace('{databaseInternalId}', $database->getInternalId(), METRIC_DATABASES_OPERATIONS_WRITES)
        ];

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow php best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue Jira

];

Authorization::skip(function () use ($dbForProject, $days, $metrics, &$stats) {
foreach ($metrics as $metric) {
$result = $dbForProject->findOne('stats', [
Query::equal('metric', [$metric]),
Query::equal('period', ['inf'])
]);
$result = $dbForProject->getDocument('stats', md5('_inf_' . $metric));

$stats[$metric]['total'] = $result['value'] ?? 0;
$limit = $days['limit'];
Expand All @@ -4331,7 +4369,7 @@ function updateAttribute(
'1d' => 'Y-m-d\T00:00:00.000P',
};

foreach ($metrics as $metric) {
foreach ([...$logsDBMetrics,...$metrics] as $metric) {
$usage[$metric]['total'] = $stats[$metric]['total'];
$usage[$metric]['data'] = [];
$leap = time() - ($days['limit'] * $days['factor']);
Expand All @@ -4347,16 +4385,16 @@ function updateAttribute(

$response->dynamic(new Document([
'range' => $range,
'collectionsTotal' => $usage[$metrics[0]]['total'],
'documentsTotal' => $usage[$metrics[1]]['total'],
'storageTotal' => $usage[$metrics[2]]['total'],
'databaseReadsTotal' => $usage[$metrics[3]]['total'],
'databaseWritesTotal' => $usage[$metrics[4]]['total'],
'collections' => $usage[$metrics[0]]['data'],
'documents' => $usage[$metrics[1]]['data'],
'storage' => $usage[$metrics[2]]['data'],
'databaseReads' => $usage[$metrics[3]]['data'],
'databaseWrites' => $usage[$metrics[4]]['data'],
'collectionsTotal' => $usage[$logsDBMetrics[0]]['total'],
'documentsTotal' => $usage[$logsDBMetrics[1]]['total'],
'storageTotal' => $usage[$logsDBMetrics[2]]['total'],
'databaseReadsTotal' => $usage[$metrics[0]]['total'],
'databaseWritesTotal' => $usage[$metrics[1]]['total'],
'collections' => $usage[$logsDBMetrics[0]]['data'],
'documents' => $usage[$logsDBMetrics[1]]['data'],
'storage' => $usage[$logsDBMetrics[2]]['data'],
'databaseReads' => $usage[$metrics[0]]['data'],
'databaseWrites' => $usage[$metrics[1]]['data'],
]), Response::MODEL_USAGE_DATABASE);
});

Expand Down Expand Up @@ -4384,7 +4422,8 @@ function updateAttribute(
->param('collectionId', '', new UID(), 'Collection ID.')
->inject('response')
->inject('dbForProject')
->action(function (string $databaseId, string $range, string $collectionId, Response $response, Database $dbForProject) {
->inject('dbForLogs')
->action(function (string $databaseId, string $range, string $collectionId, Response $response, Database $dbForProject, Database $dbForLogs) {

$database = $dbForProject->getDocument('databases', $databaseId);
Comment on lines +4426 to 4428

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functional High

The collection usage endpoint dereferences $database->getInternalId() before verifying the database exists, so add a database emptiness check before building collection paths.

Suggested fix
    ->action(function (string $databaseId, string $range, string $collectionId, Response $response, Database $dbForProject, Database $dbForLogs) {

        $database = $dbForProject->getDocument('databases', $databaseId);

        if ($database->isEmpty()) {
            throw new Exception(Exception::DATABASE_NOT_FOUND);
        }

        $collectionDocument = $dbForProject->getDocument('database_' . $database->getInternalId(), $collectionId);
Prompt for AI assistance

Copy the prompt below and paste it into ChatGPT, Claude, or any LLM:

You are an expert php developer with deep knowledge of security, performance, and best practices.

### Context

File: app/controllers/api/databases.php
Lines: 4426-4428
Issue Type: functional-high
Severity: high

Issue Description:
The collection usage endpoint dereferences `$database->getInternalId()` before verifying the database exists, so add a database emptiness check before building collection paths.

Current Code:
    ->action(function (string $databaseId, string $range, string $collectionId, Response $response, Database $dbForProject, Database $dbForLogs) {

        $database = $dbForProject->getDocument('databases', $databaseId);
        $collectionDocument = $dbForProject->getDocument('database_' . $database->getInternalId(), $collectionId);

---

### Instructions

1. Fix the issue described above
2. Maintain the exact indentation and code style from the original
3. Follow php best practices and language-specific idioms
4. Ensure the fix addresses the root cause, not just the symptoms
5. Add brief inline comments explaining the fix if needed

### Constraints

- Do not change functionality beyond fixing the identified issue
- Preserve existing variable names and function signatures unless they are part of the problem
- Ensure the fix is production-ready

---


Like Dislike Create Issue Jira

$collectionDocument = $dbForProject->getDocument('database_' . $database->getInternalId(), $collectionId);
Expand All @@ -4401,17 +4440,14 @@ function updateAttribute(
str_replace(['{databaseInternalId}', '{collectionInternalId}'], [$database->getInternalId(), $collectionDocument->getInternalId()], METRIC_DATABASE_ID_COLLECTION_ID_DOCUMENTS),
];

Authorization::skip(function () use ($dbForProject, $days, $metrics, &$stats) {
Authorization::skip(function () use ($dbForLogs, $days, $metrics, &$stats) {
foreach ($metrics as $metric) {
$result = $dbForProject->findOne('stats', [
Query::equal('metric', [$metric]),
Query::equal('period', ['inf'])
]);

$result = $dbForLogs->getDocument('stats', md5('_inf_' . $metric));
$stats[$metric]['total'] = $result['value'] ?? 0;

$limit = $days['limit'];
$period = $days['period'];
$results = $dbForProject->find('stats', [
$results = $dbForLogs->find('stats', [
Query::equal('metric', [$metric]),
Query::equal('period', [$period]),
Query::limit($limit),
Expand Down
Loading